vault: fix the snapshot token, which was built to die monthly - #607
Merged
Merged
Conversation
Fifteen days with no Vault snapshot - last one Sep 8, failing nightly with 403 invalid token. Three stacked defects, all in the six-line one-time recipe at the top of backup-vault.yml. 1. No -orphan, so the token was a child of token-admin - which docs/OPS.md and 1.11 say is deliberately allowed to lapse. Vault revokes children when the parent expires. token-admin is minted at 720h; one issued around 08-09 expires 09-08, and the last good snapshot is 09-08. 2. -period=87600h was never honoured. Vault caps period at the effective max_ttl of 768h and warns at creation. A comment promising ten years described a 32-day token. 3. Nothing renewed it. A periodic token lives one period at a time, so vault token renew -self now runs before every snapshot and fails the unit if it cannot. The monitoring was not the failure: LabBackupUnitFailed fired nightly and ntfy delivered it, to a phone that had been stolen. One subscriber, so losing the device lost the channel. Records 2.13's second occurrence - a live token pasted while diagnosing this, because the command I suggested printed it. The fault is in suggesting a command that can print a secret, not in the pasting; the header recipe now uses -field with a redirect. Also fixes a second marker bug in backlog-audit.py: markers match as substrings, so NOT YET APPLIED contains APPLIED and the tool read 1.14's heading as a completion claim. Strips 'not yet <word>' before matching; verified against seven headings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fifteen days with no Vault snapshot - last one Sep 8, failing nightly with 403 invalid token. Three stacked defects, all in the six-line one-time recipe at the top of backup-vault.yml.
No -orphan, so the token was a child of token-admin - which docs/OPS.md and 1.11 say is deliberately allowed to lapse. Vault revokes children when the parent expires. token-admin is minted at 720h; one issued around 08-09 expires 09-08, and the last good snapshot is 09-08.
-period=87600h was never honoured. Vault caps period at the effective max_ttl of 768h and warns at creation. A comment promising ten years described a 32-day token.
Nothing renewed it. A periodic token lives one period at a time, so vault token renew -self now runs before every snapshot and fails the unit if it cannot.
The monitoring was not the failure: LabBackupUnitFailed fired nightly and ntfy delivered it, to a phone that had been stolen. One subscriber, so losing the device lost the channel.
Records 2.13's second occurrence - a live token pasted while diagnosing this, because the command I suggested printed it. The fault is in suggesting a command that can print a secret, not in the pasting; the header recipe now uses -field with a redirect.
Also fixes a second marker bug in backlog-audit.py: markers match as substrings, so NOT YET APPLIED contains APPLIED and the tool read 1.14's heading as a completion claim. Strips 'not yet ' before matching; verified against seven headings.